[diffs] Byte Arena Experiments#775
Draft
clemg wants to merge 10 commits into
Draft
Conversation
* Remove history coalesce
* Fix selction/crate not updated when do "redo" command
* Remove visualColumns.ts
* Move editor ts files
* Refactor textarea buffer
* Rename `EditSnippet` type to `TextareaSnapshot`
* Remove `Editor` component, introduce the `Editor` class for `File` component
* Update demo
* Update editor constants to set text and background color to transparent
* Rewrite rerender logic
* Format
* Remove dead code
* Fix caret postion on empty line
* Improve `renderSelectionRange` performance by using cached DOM elements
* Support range selection in textarea
* Improve rerender performance
* Use piece table data sturcture for the text document
* refactor
* Add public `setSelection` method for the `Editor` class
* Add `FileContentsWithLineOffsets` interface and update related components to support line offsets and line count. Refactor file handling to utilize computed line offsets for rendering and iteration.
* Add `updateRenderCacheAt` method to `FileRenderer` and `File` classes for improved rendering. Refactor theme handling in `Editor` to utilize a dedicated method for color map retrieval.
* Refactor file iteration logic by removing `iterateOverFile` utility and replacing it with direct loops in `VirtualizedFile` and `FileRenderer` components. Update line offset computation to exclude trailing newlines in multi-line files while maintaining correct line counts. Enhance tests to validate line counting behavior.
* Remove EOF field
* Remove text length fields from HistoryEntry and related test cases in EditHistory
* Rename class `EditHistory` to `EditStack`
* Refactor EditStack and PieceTable to use a unified text slice interface.
* Refactor PieceTable and TextDocument to improve line offset handling and remove unnecessary EOL trimming logic.
* Refactor `Editor` to utilize new dirty line resolution logic, enhancing performance and accuracy in line tracking.
* Fix multi-cursor textarea sync
* Refactor Editor rendering logic for improved performance and reduce direct DOM manipulation.
* Add grammer cache
* Enhance line position caching in Editor for improved performance and accuracy.
* Refactor indentation handling in Editor and remove unused utility function for improved clarity and performance.
* Fix testing types
* Improve performance of the `getCharacterX` method
* Improve caching mechanism for enhanced performance.
* Add maxEntries feature to EditStack for managing undo history size
* Refactor
* Enhance PieceTable and TextDocument to trim line endings in getLineText method, improving text handling consistency. Update related tests for accuracy.
* Refactor
* Add `BackgroundTokenzier` class
* Improve performance
* Fix hightlight bug
* Add `--diffs-bg-caret` css property
* Fix input
* Fix selection range rendering
* Fix prebuildStateStackCache funciton
* Update `TOKENIZE_MAX_LINE_LENGTH` to 10,000
* Add `DiffsEditor` interface
* Fix `lineAnnotations` argument on `triggerEdit` invoke
* Refactor editor edit method to accept onChange callback directly and update demo to log file changes
* Clean up
* typo
* Refactor BackgroundTokenizer to use message-based scheduling.
* Refactor editor focus handling by removing redundant event listeners and updating CSS selectors for caret visibility.
* Refactor
* Fix `toTextareaSelectionDirection` function
* Refactor
* Update `DiffsEditor` types
* Add line annotation handling
* Add documentation for `hasVisibleLineAnnotation` function.
* Get rid of enum
* Clean up
* Refactor
* Update editor CSS
* Support text wrap
* Clean up
* Fix line y/wrap cache
* Fix line cache
* Copies leading indentation onto the new line after Enter
* Focus textare after undo/redo
* Move multi-selection functions to editorSelection module
* Add support for handling leading indentation deletion in applyTextChangeToSelections
* Fix selection glitch bug
* Add extendSelection command
* Fix `focusTextare` function
* Fix `resolveTextareaChange` function
* Remove unnecessary target check in mouseup event listener in Editor class
* Fix textarea selction direction
* Fix selection bg color for safair
* Clean up
* Fix shift select
* Refactor
* Refactor
* Fix shift select delay
* Coalesce edit stack entries for simple typing or backspace operations.
* Add Support forward-delete coalescing for edit history
* docs: add docs for editStack module
* Refctor
* Fix 'documentStart' and 'documentEnd' commands
* Rewrite selection handle logic
* Fix shouldCoalesceEditStackEntry function
* Update demo
* Add `removeEditor` for File component
* Add react api
* Clean up
* Update demo app
* Refactor useFileInstance to remove redundant editor cleanup logic
* Fix `computeLineOffsets` function
* typo
* Update editor style
* Fix `getOrCreateLineOffSets` method
* Refactor line count and annotation handling in File component; remove hasVisibleLineAnnotation utility
* Fix lines deletion crocss virtul viewport
* Remove `normalizeSelectionsForDocument` function
* Fix `edit` function
* Add editor sub-module
* Use `contenteditable` model
* Fix line wrap
* Fix wrap line
* Fix selection on mobile
* Update editor style
* Fix resize handling
* Add editor overlay layer
* Cleanup
* Add `DiffsEditableComponent` types
* Fix `VirtualizedFile` component
* Update `DiffsEditableComponent` type
* Add editor demo
* Fix slection rendering
* Update editor demo app
* Fix VirualizedFile component
* Update editor demo app
* Fix some selection bugs
* Update demo app
* Refactor findNextNonOverlappingSubstring method into PieceTable and TextDocument
* Refactor
* feat: Implement line jump
* Fix selection rendering when scrolling
* Improve tokenzier performance
* feat: simple search pannel
* Update editor demo app
* Fix jump
* Update search UI
* Add lag radar
* Fix virtualizer
* Fix render range after typing
* Fix editor tokenzier cache
* Fix search input focus
* Update log rader position
* Improve piece table performance
* Refactor
* Add lag radar
* Fix line count for empty documents
* Fix offscreen lines flush
* Introduce gutter width tracking
* refactor
* Refactor
* fix import
* Add 'expandSelectionDocStart' and 'expandSelectionDocEnd' commands
* Fix buffer height
* Add matches text for the search pancel
* Disable preious/next icon when no matches
* Update style.css
* feat: Support `quiteEdit` action
* Update edtior demo app
* Refactor
* Update demo app
* Update demo app
* Fix girdRow when render quick edit UI
* Move testing files
* Clean up
* Add searchPanel.ts
* Fix expandCollapsedSelectionToWord to match when the cursor is immediately touching one of the word's boundaries
* clean up
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Fix typo
* Clean up searchPanel and quickEdit when swith file
* Rebase to beta-1.2
* Fix selection after clean up quick edit widget
* Fix virtual buffer
* Fix `updateWindowSelection` method of Editor class
* Fix render range when typing new line at the end of the file
* Fix buffer when adding large lines
* [editor] Support 'deleteHardLineForward' input
* Add `insertTranspose` input
* Move `change` handler to options
* Update css
* Merge beta-1.2 changes
* Fix emply line rendering
* Add search settings UI
* Merge branch 'main' into editor
* Support FileDiff component
* Update `DiffsEditableComponent` interface
* Fix `getSelectionAnchor` function
* Fix text measurement for emoji
* Increase delay for diff rendering in FileDiff component
* Update types
* Add unit testings for text measue functions
* Clean up dirty render cache
* Fix `lineAnnotations` re-rendering
* Disable gutter utility when editing
* Add global css
* Fix scrollToLine method
* Refactor selection handling in Editor class to initialize selections properly and streamline rendering logic
* Fix diffs components
* Allow to create selection from gutter interaction
* Fix focus
* Fix browser compatibility
* Support dual themes
* Fix selection bugs
* refactor
* Add `Metrics` class
* Clean up
* Fix wrap selection rendering on safari
* Add `QuickEditContext` types
* Fix caret scroll margin when search panel is on
* Refactor search panel widget
* Fix selection position
* Update react components
* Update search panel CSS
* Fix quick edit
* Add editor docs
* Fix react hooks for editor
* Update editor demo component
* Update Quick Edit docs
* Update `diffStyle` and `expandUnchanged` options when editing
* demo: remove editor route
* Update docs
* Update docs
* Update examples
* Reset selection when 'Esc' key pressed
* Fix selection focus
* Add 'enable edit' shortcut('e')
* Handle the arrow key events to scroll to the cursor position manually
* Merge of overlapping selections
* Handle cursor moving events
* Fix scroll margin top
* Add debug logging option to Editor class
* Fix selection bugs
* Fix selection renering for unified `FileDiff`
* Reset ignore selection change flag on mouse up event
* Clean up
* fix bun.lock
* Add editor theme style
* Refactor
* Fix react types
* Fix last line index calculation
* Update condition for marking DOM dirty in VirtualizedFile component
* Throw if someone is trying to edit with no editor instance
* Update `mergeFileDiffOptions` function
* `lineOffsets` -> `lines`
---------
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
`additionLines`/`deletionLines` change from `string[]` to `DiffLines`: a plain
data object holding a file's lines as one UTF-8 byte arena plus an offset table,
decoded on demand via `lineAt` / `joinLines`. On a huge diff (linux v6..v7,
~22.8M lines across ~77k files) this avoids tens of millions of tiny `String`
objects, so the V8 heap drops ~33% on that compare and the parser is faster: it
no longer encode+decode-detaches every line, it encodes once on seal and decodes
only the visible (virtualized) lines.
It is plain data on purpose, so it survives structured clone (the highlight
worker), `structuredClone`, and IndexedDB without a revive step (no class, no
prototype to drop). `.length` stays a field, so the many `.length` consumers are
unchanged; only content reads migrate (`x[i]` -> `lineAt(x, i)`,
`x.join('')` -> `joinLines(x)`). Per-file offsets use the smallest int width that
fits the file. A file with a lone surrogate keeps exact strings as a fallback,
and merge-conflict diffs keep plain strings (no encode) so their parse stays at
parity. The parsed model is byte-identical to before (snapshot + content-hash).
Adds diffLines.test.ts (arena round-trip, multibyte, emoji-keeps-arena, lone-surrogate fallback, BOM, offset-width, plainLines, joinLines, isWellFormed) and a withPlainLines snapshot converter so the existing parsed-model snapshots assert byte-identical line content.
The byte-arena type change makes additionLines/deletionLines a DiffLines, so the editor's FileDiff whole-side accessors (getDeletionFile/getAdditionFile) read them with joinLines(...) instead of .join('').
|
@clemg is attempting to deploy a commit to the Pierre Computer Company Team on Vercel. A member of the Team first needs to authorize it. |
Member
|
Hey sorry for the delays on this, was AFK for a couple days on a trip, but will dig more into this, this week! |
5652895 to
667717a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Changing the way we represent patch storage from a
string[]to a byte arena (a contiguous byte array), which uses less memory the bigger the PR gets.For full explanation and details, see: #760
Note that I didn't update any documentation yet
Motivation & Context
About half of the time, I was getting OOMs crashes on huge PRs like the linux v6..v7 comparison (which is not too crazy given the size of the patch), but still annoying. I think this other way of representing the lines is more efficient and can help either rendering bigger diffs on good hardware, or just normal diff on older hardware
Type of changes
first discussed with the dev team and they should be aware that this PR is
being opened
You must have first discussed with the dev team and they should be aware
that this PR is being opened
Checklist
contributing guidelines
bun run lint)bun run format)bun run diffs:test)How was AI used in generating this PR
The tests have been fully generated by opus 4.8
Related issues
See: #760